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(54) Method and apparatus for hemispheric imaging 



(57) A system for electronic imaging of a hemispher- 
ic field of view includes, a camera for receiving optical 
images of the field of view and for producing output data 
corresponding to the optical images. The camera in- 
cludes an optical assembly for producing images 
throughout a hemispheric field of view for optical convey- 
ance to an imaging device or photographic film. The op- 
tical system assembly has lens components that selec- 
tively emphasize the peripheral content of the hemi- 
spheric field of view. An electronic imaging device within 
the camera or a film-to-digital data conversion system 
provides digitized output signals to input image memory 
or electronic storage devices. A transform processor se- 
lectively accesses and processes the digitized output 
signals from the input image memory according to us- 
er-defined criteria and stores the signals in output image o 
memory. The signals in the output image memory can 
then be displayed according to the user-defined criteria. 
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Description 

The present invention relates generally to visual im- 
aging systems, and more particularly to visual irriaging 
systems and techniques which provide useful electronic 
manipulation of wide angle hemispheric scenes. 

The collection, storage, and display of large areas 
of visual information can be an expensive and difficult 
process to achieve accurately. With the recent increased 
emphasis on multimedia applications, various methods 
and apparatus have been developed to manage visual 
data. A unique class of multimedia data sets is that of 
hemispheric visual data. Known multimedia methods 
and apparatus attempt to combine various multimedia 
imaging data, such as still and motion (or video) images, 
with audio content using storage media such as photo- 
graphic film, computer diskettes, compact discs (CDs), 
and interactive CDs. These are used in traditional multi- 
media applications in various fields, such as entertain- 
ment and education. Non-multimedia applications also 
exist that would employ hemispheric visual data, such 
as in security, surveillance, unmanned exploration, and 
fire and police situations. However, as will be described 
below, the known methods and apparatus have certain 
limitation in capturing and manipulating valuable infor- 
mation of hemispheric scenes in a rapid (ie real-time) 
and cost-effective manner. 

One well-known multimedia technique is used at 
theme parks, wherein visual information from a scene is 
displayed on a screen or collection of screens that covers 
almost 360 degrees field of view. Such a technique un- 
fortunately results in the consumption of vast quantities 
of film collected from multiple cameras requires special- 
ly-designed carriages to carry and support the cameras 
during filming of the scene.and necessitates synchroni- 
zation of shots duringcapture and display. The technique 
is also limited in that the visual image cannot be obtained 
with a single camera nor manipulated for display, eg pan, 
tilt, zoom, etc., after initial acquisition. Hence, this tech- 
nique, while providing entertainment, is unable to fulfil 
critical technk^al requirements of many functional appli- 
cations. Other known techniques for capturing and stor- 
ing visual information about a large field of view (FOV) 
are described in US Patent Nos. 4,125,862; 4.442.453 
and 5,185,667. In US Patent No. 4,125.862, a system is 
disclosed that converts single information from a scene 
into digital form, stores the data of the digitized scene 
serially in two-dimensional format, and reads out the 
data by repetrtive scan in a direction orthogonally related 
to the direction in wtiich the data was stored. US Patent 
No. 4,442,453 discloses a system In which a landscape 
is photographed and stored on film. The film is then de- 
veloped, with display accomplished by scanning with 
electro-optical sensors at "near real-time" rates. These 
techniques, however, <jo not provide instant visual image 
display, do not cover the field of view required for desired 
applications (hemispheric or 180 degrees field of view, 
do not generate visual image data in the format provided 



by the technk:;ues of this Invention, and are also not eas- 
ily manipulated for further display, eg-pan, tilt, etc. 

The technique disclosed In the U.S. Patent No. 
5,185,667 overcomes some of the above-identified 
5 drawbacks in that it is able tocapture a near-hemispheric 
field of view, correct the image using high speed circuitry 
to form a normal irriage. and -electronically manipulate 
and display the image at real-time rates. 

For many hemispheric visual apptteations, however, 
10 a system of the type described by the 667 patent has 
limitations in obtaining sufficient information of crKical 
and useful details. This is partrcularty true when the cam- 
era is oriented with the central axis of the lens perpen- 
dicular to the plane bounding the hemisphere of acqui- 
ts sition (i.e. lens pointing straight up). In such applications, 
the majority of critical detail in a scene is contained in 
areas of the field along the horizon and little or no useful 
details are contained in central areas of the field kx:ated 
closer to the axis of the lens (the horizon being defined 
20 as the plane parallel to the image or camera plane and 
perpendicular to the optical axis of the imaging system). 
For example, in sun/eillance, the imaging system is 
aimed upward and the majority of the critical detail in the 
scene includes people, buildings, trees, etc. most of 
25 which are located within only a few degrees along the 
horizon (i.e., this is the peripheral content). Also, in this 
example, although the sky makes up the larger central 
area of the view, it contains little or no useful information 
requiring higher relative resolutbn. 
30 To obtain sufficient detail on the critical objects in the 
scene, the technique should be able to differentiate be- 
tween the relevant visual information along the horizon 
and the remaining visual information in the scene in order 
to provide greater resolution in areas of higher impor- 
ts tance. The system described by the 667 patent does not 
differentiate between this relevant visual information 
contained along the horizon and the remaining visual in- 
formation in this scene. Thus, it fails to yield a sufficient 
quality representation of the critical detail of the scene 
40 for projected applications. 

Instead, techniques <iescribed above concentrate 
on obtaining, storing, and displaying the entire visual In- 
formation in the scene, even when portions of this infor- 
mation are not necessary or useful. To obtain the 
4$ near-hemispheric visual information, such techniques 
require specific lens types to map image information in 
the fieW of view to an image plane (where either a pho- 
tographic film or electronic<letector or imager is placed). 
Known examples of U.S. PatentNo. 5.185,667 and U.S. 
60 Patent No. 4.442.453 respectively use a fish-eye lens 
and a general wide-angle lens. As these lenses map in- 
formation of a large fiekd without differentiation between 
thecentral andperipheral areas, information from the pe- 
riphery will be less fully represented in the image plane 
55 than from the central area of acquisition. 

U.S. Patent No. 4, 170.400 describes a wide-angle 
optk;al system enptoying a fibre optic bundle that has 
differing geometrk: shapes at the imaging «nds. Al- 
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though this is useful in itseH (or collecting and reposition- 
ing image data, bending of light is a natural characteristic 
ot optical fibres and not exclusive to that patent. Further, 
4,1 70,400 employs a portion of a spherical mirror to'gath- 
er optical information, rendering a very reduced subset s 
of the periphery in the final imaging result. This configu- 
ration is significantly different from the multi-element lens 
combination described in the present invention. 

Imperfections in the image representation of any 
field inherently result from the nature of creating an im- 
age with any spherical glass (or plastic) medium such as 
a lens. The magnitude of these imperfections increases 
proportionally to the distance a point in the field is from 
the axis perpendicular to the optical imaging system. As 
the angle between the optical axis and a point in the field »5 
increases, aberrations of the corresponding image in- 
crease proportional to this angle cubed. Hence, aberra- 
tions are more highly exaggerated in the peripheral are- 
as with respect to more central areas of a hemispheric 
innage. 

Although the lens types above achieve a view of a 
large field, the valuable content from the peripheral areas 
lacks in potential image quality (resolution) mapping be- 
cause the imaging device and system does not differen- 
tiate between these areas and the central areas of less 
valuable detail. Often, the difference between the imag- 
ing capabilities between the two areas is compensated 
for by using only the central portion of a lens to capture 
the scene ("stopping the lens down*). This works in effect 
to reduce the image quality of both areas such that the 3p 
difference in error is a lesser percentage of the smallest 
area even the central area can resolve. Simultaneously, 
this compensation technique further degrades the per- 
formance of the lens by limiting the amount of light which 
Is allowed to enter the lens, and thus reducing the overall 35 
intensity of the innage. 

More typically, the peripheral content imaged by a 
conventional lens is so degraded in comparison with the 
central area that the lens allows for only a minimal area 
of the periphery to be recorded by the film or electronic 
imager. As a result of these "off-axis" aberrations inher- 
ent to large fields, the relevant information of the horizon 
in the scene can be underutilized or worse yet — lost. 

Another limitation in patent 5,185,667 is its organi- 
zation for recording only views already corrected for per- <s 
spective. The nature of that methodology is that the spe- 
cific view of interest must be selected and transformed 
prior to the recording process. The result is that no ad- 
ditional selection of views can be accomplished after the 
storage process, reducing system flexibility from the us- 50 
er's perspective. 

Hence, there is a demand in the industry for single 
camera imaging systems that efficiently capture, store, 
and display valuable visual information within a hemi- 
spheric field of view containing particularly peripheral ss 
content, and that allow electronic manipulation and se- 
lective display of the image post-acquisition while mini- 
mizing distortion effects. 



The present inverHion provides a new and useful vis- 
ual innaging system that emphasizes the peripheralcon- 
tent of a hemispheric field of view using a singtecamera. 
The captured visual information can t)e stored as a single 
image using conventional chemical-based <film) or elec- 
tronic data storage techniques. The invention allows us- 
er-selected portions of a hemispheric scene to-be .elec- 
tronically manipulated arud displayed from the stored vis- 
ual database in a real-time and cost-effective nr^nner. 

According to the present invention there is provided 
a system for electronic imaging and manipulalion of a 
hemispheric field of view, comprising: 

a camera for receiving optical images of a hemi- 
spheric field of view and for producing output signals or 
affecting photographic film-based materials correspond- 
ing to the optical images; 

an optical imaging device associated with the cam- 
era for producing optical images throughout the hemi- 
spheric field of view for optical conveyance to the cam- 
era, the optical system having a configuration adapted 
to capture and enhance an image of peripheral regions 
of the hemispheric field of yiew; 

an image processing device associated with the 
camera and the optical imaging system for receiving op- 
tical images from the lens and forproviding digitized out- 
put signals representative of the received optical images; 

Input image nnemory for receiving the digitized out- 
put signals f-rom the imager device and for storing the 
digitized output signals; 

image transform processing circuitry for selectively 
accessing and processing the digitized output signals 
from the input image memory according to user^ef ined 
criteria; 

output image memory for receiving the processed 
signals from the image transform processor, and 

an output -display or recording device connected to 
the output image memory for recording the signals in the 
output image memory. 

Suitably the optical system magnifies a portion ot the 
field of view within a range of approximately one XoAox- 
ty-five degrees above the horizon of a-captured image 
of a hemispheric field of view, and preferably the optical 
system has a configuration which images a peripheral 
portion of the hemispheric scene onto at least 50% of an 
imaging area of an imaging device. 

The optical system may include a wide field multi-el- 
ement lens positioned todirect an image^o a tight 4-rans- 
mitting fibre array, part icularty one wherein the fibre array 
is geometrically arranged to have a generally annular in- 
put end and a generally rectangular output ^nd, or 
wherein the-fibre array is geometrically arranged to have 
a generally annular input end and a -generally circular 
output end. Preferably the fibres of the fibre array have 
an imaging accuracy on the order of three microns. The 
system Is most suitably one wherein the -focal length of 
the wide field lens is scaled to match a desired magnifi- 
cation of the peripheral field of view. 

The system is suitably configured such that the op- 
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tical imaging system includes a cotour-aberrated multi- 
ple element wide field lens In combinatbn with a gradient 
index hemispheric lens, in particular wherein the innage 
processing device is an electronic imager, and preferably 
of the type wherein the image processing device is a s 
charge-coupled device. 

The system of the invention is preferably one where- 
in the image processing device is photographic film, and 
more preferably further includes a capture and conver- 
sion device thatijonverts the optical images on the pho- io 
tographic film into digital output signals for input to the 
input image memory. A suitable capture and conversion 
device is an NTSC4o-digital converter. 

Data from thecapture and conversion device is pref- 
erably input to a source image frame t^uffer. in particular is 
wherein the image transform processing circuitry <;om- 
prises an image re-sampling sequencer; the image 
re-sampling sequencer preferably controls address se- 
quencing of pixels in the source image frame buffer, and 
the image transform processing circuitry suitably further 20 
comprises row and column warp-engines. 

The image transform processing circuitry may fur- 
ther comprise a warped image buffer connected to the 
warp engines and to which an output of the source image 
frame buffer is input. The image transform processing 2S 
circuitry may further comprise a look-up table containing 
transformation parameters for perspective correction of 
hemispheric field of view images. Such image transform 
processing circuitry preferably further comprises an in- 
terpolation coefficient buffer connected to the look-up ta- 30 
b\e to update the transformation parameters, and con- 
nected to a multiply/accumulate unit. 

In the system of the present invention the output dis- 
play may be worn on or attached to a viewer's head, so 
that the orientation of the viewer with respect to the im- 55 
aged surroundings is electronically conveyed and inter- 
preted as user<:ontrots. 

According to a second embodiment of the present 
invention there is provided a system for imaging and ma- 
nipulation of a hemispheric field of view comprising: 40 

(I) an optk:al system for producing anoptfcal image 
of a hemispherteal field of view, the optical system 
having a central lens axis coaxial with acentral axis 
of the hemispherical field of view and a configuration 
which emphasizes through differential magnification 
of peripheral content of the hemispherical field of 
view, the optical image having a defined usable 
Image area when projected onto a plane; 



iW) a camera optically coupled to the optkjal system 
for receiving optical images of the hemispherical 
field of view and for producing an output correspond- 
ing to the optcal images; 

<lii) input image memory coupled to the camera for 
receiving and storing the output from the camera; 
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(iv) a processor coupled to the input image memory 
for selectively accessing from the input image mem- 
ory and processing output from the camera, the 
processor for transforming the output from the cam- 
era according to userdefined criteria into a proces- 
sor output which has a peripheral image content 
emphasis which differs from that of the optical sys- 
tem; 

(v) output image memory coupled to the processor 
for receiving and storing the processor output; and 

<vl) an output device coupled to the output image 
memoiy for rendering the stored processor output 
into a visual image transformed from the optrcal 
image in accordance with the user defined criteria. 

The optical system suitably has a configuration that 
images the peripheral portion of a hemispherical scene 
onto a portion of the optical image usable image area 
which is at least about fifty percent of the optical image 

usable image area. 

Preferably the optical system has a configuration 
that images the peripheral portion of a hemispherical 
scene onto a portion of the optical image usable image 
area which ranges from about fifty percent to about nine- 
ty percent of the optical image usable Image area. 

Alternatively the optical system may have a config- 
uration which enr>phasizes through differential magnifica- 
tion at least a portion of that content of the hemispherfcal 
field of view which lies between a base plane of the hem- 
ispherical field of view and a right cone defined about 
said central lens axis, said cone having an included an- 
gle of forty five degrees between saki central lens axis 
and a line generating said cone and with said line gen- 
erating said cone passing through the point of intersec- 
tion of said central lens axis with sakJ base plane. 

More preferably the optfcal system has a configura- 
tion that images the peripheral portion of a hemisphercal 
scene onto a portk)n of the optrcal image usable image 
area which is no more than about ninety percent of the 
optkjal image usable image area, and may suitably be 
an optk:al system conr^prising: 

(I) a wide field multi-element lens; and acoordinated 
fibre array geometrcally arranged to have a gener- 
ally annular input end and a generally rectangular 
output end; or 

(ii) a wide field multi-element lens; and acoordinated 
fibre array geometrically arranged to have a gener- 
ally annular input end and a generally annular output 
end. 

<iii) a cotour-aberrated wide fiekJ lens; and a gradient 
index hemispheric lens. 

The system may suitably further comprise an imager 
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device interposed between the camera and the input im- 
age memory for receiving the output from the camera 
and for generating a digitized output, and further wherein 
the input image memory, the processor, and said output 
image memory receive, store and process the digitized 
output. Preferably the camera comprises a light sensitive 
electronic image capture element and the imager device 
comprises an electronic digitizer circuit. More preferably 
the system further comprises photographic film for pho- 
tographically capturing said output from the camera and 
further wherein the imager device comprises a capture 
and conversion device which converts the photographi- 
cally captured output from the camera into the digitized 
output. 

The system of this embodiment is preferably one 
wherein the processor comprises image processing and 
warping circuitry for transforming the. output from the 
camera according to user defined criteria into a proces- 
sor output which has a peripheral content emphasis 
which differs from that of the optical system, and more 
preferably the image processing and warping circuit 
comprises an arithmetic logic unit (ALU), and further the 
processor comprises a storage memory device coupled 
to the ALU for receiving and storing control programs for 
determining the functions served by the ALU, and a con- 
trol program stored in the storage memory device and 
accessible to the ALU for controlling theoperation of the 
ALU to transform the output from the camera according 
to user defined criteria into a processor output which has 
a peripheral/central content emphasis which differs from 
that of the optical system. 

The processor suitably generates processor output 
which represents a selected portion of the hemispherical 
field of view. 

The system aptly includes an output device compris- 
ing a visual display device for displaying a visual image 
closely adjacent the eyes of an observer. 

In the preferred embodiment of the invention the 
processor generates processor output which represents 
a selected portion of the hemispherical field of view, and 
the system further comprises: 

sensors for detecting the orientation of the head or 
eyes of an observer, and 

a link between the sensors to the processor for 
modifying the generation of processor output as a func- 
tion of the detected orientation of the eyes of an observer. 

In a further preferred embodiment of the present in- 
vention there is provided a method for electronically cap- 
turing, storing, and manipulating a hemispheric field of 
view, comprising the steps of: 

providing an optical system having a configuration 
that enhances the peripheral portion of the field of view, 

capturing the hemispheric field of view with the pe- 
riphery-enhancing optical system and imaging the field 
of view onto an imager device by enhancing the periph- 
eral field of view, 

storing the captured image as a single image, 

selectively accessing a portion of the stored image 



according to user-defined criteria. 

transforming the stored image so that the stored 
image can be displayed as a perspective-iCOFrect image, 
displaying the perspective-correct image in a us- 
5 er-defined format. 

Preferably the transforming step comprises manip- 
ulating the peripheral -enhanced image into a perspec- 
tive-correct innage. 

In this preferred emlx>diment the periphery-enhanc- 
ro ing optical system optionally selectively magnifies visual 
content within an arc of between 10 and 45 degrees up 
from the horizon in a hemispheric field of view onto the 
imager device. 

The method may also Include a storage step com- 
is prising storing the captured image onto photographic film 
and the step of selectively accessing a portion of the 
stored image comprising converting the image from the 
photographic film into digital output format. The storage 
step preferably comprises storing the captured image in 
20 electronic storage. 

In a further embodiment of the present invention 
there is provided a method for electronically manipulat- 
ing a hemispheric scene having an.er>harK:ed peripheral 
field of view stored as an image on photographic film. 
25 comprising the steps of: 

converting the image on the photographic film into 
electronic output signals, 

selectively accessing a portion of the output sig- 
nals according to user-defined criteria. 
30 transforming the accessed portion of the output 

signals by manipulating the (Deripheral-enhanced field of 
view so that the stored imagecan be displayed as a per- 
spective-correct image, 

displaying the perspective-correct image in the us- 
35 er-defined format. 

In yet a further embodiment of the present invention 
there is provided a method ^or generating image se- 
quences over physical distances from a reduced set of 
enhanced-periphery hemispheric images, wherein inter- 
ne mediate images are interpolated from a combination of 
adjacent hemispheric images, the method comprising 
the steps of: 

defining how many intermediate images are need- 
ed to satisfy the simulation of motion-video or film-based 
45 movie cameras; 

interpolating needed internrtediate Images from a 
combination of adjacent hemispheric images; 

identifyingor electronically taggir»g interpolated im- 
ages, then storing or recording them for later introduction 
so to the image processir>g subsystem; 

loading original and interpolated image data sets 
In appropriate sequence to the image processing sub- 
system at rates sufficient to simulate real-time nnotion 
video, and 

55 displaying the -perspective-correct image in the us- 

er-defined format. 

In yet a further embodiment of the present invention 
there is provided a system for«lectronic manipulation of 
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a hemispheric scene, comprising: 

a camera imaging system for imaging a hemi- 
spheric field of view; 

an optical system associated with the<:amera im- 
aging system tor producing the optica! images through- 
out the field of view for optical conveyance to the camera 

imaging system; 

an imager device associated with the camera for 
receiving the optical images from the lens and for pro- 
viding digitized output signals; 

input image memory for receiving the digitized out- 
put signals from the imaging device and for storing the 

digitized output signals; 

image transform processor circuitry for selectively 
accessing and processing the digitized output signals 
from the input image memory according lo user-defined 
criteria; 

output image memory for receiving the processed 
signals from the image transform processor means; 

an output display device or electronic recording de- 
vice connected to the output image memory for display- 
ing or recording the signals in the output image memory 
according to user-defined criteria; 

the optical system having a configuration that em- 
phasizes the peripheral content of the field of view of a 
hemispheric scene as compared to the central content, 
such that the imager device receives magnified optical 
images of the peripheral portion of the hemispheric field 
of view. 

The visual imaging system of the invention involves 
either a stilt imageor a moving picturecamera. electronic 
or othenwise. having a lens with enhanced peripheral 
content imaging capabilities. The lens provides an en- 
hanced view of the valuable information in the scene's 
periphery by imaging a field to the image plane such that 
the ratio of the size of the smallest detail contained within 
the periphery of the scene to the size of the smallest re- 
solving pixel of an image device is increased. For this to 
be accomplished, the peripheral content must map to a 
larger percentage of a given image detector area and, 
simultaneously, the mapped image of the central area of 
the scene must be minimized by the lens so that it does 
not interfere with the peripheral content now covering a 
wider annulus in the image plane. Information in the im- 
age plane is then detected by an imager device (either 
a photographic film or electronic imager or video detector 
array). The detected information of the entire hemispher- 
ic scene is then stored as a single image in memory using 
traditional methods. 

When a portion of the scene is to be displayed, the 
image information relating to the relevant portion of the 
scene is instantaneously retrieved from memory. A 
transform processor subsystem electronically manipu- 
lates the scene for display as a perspective-correct im- 
age on a display device, such as a conventional nrvonitor 
or TV, as if the particular portion of the scene had been 
photographed with a conventional camera. The trans- 
form processor subsystem compensates for the distor- 



10 

tion or difference in magnification between the central 
and peripheral areas of the scene caused by the lens by 
applying appropriate correction criteria to bring the se- 
lected portion of the scene into standard viewing fomnat. 

5 The transform processor subsystem can also more fully 
compensate for any aberrations of the enhanced periph- 
eral image because of the image's improved resolution 
as it covers a larger portion of the image device (in- 
creased number of pixels used to detect and measure 

10 the smallest detail in the periphery image). More pixels 
equates to more measurement data, hence more accu- 
rate data collection. 

The stored image can also be manipulated by the 
transform processor subsystem to display an opera- 

16 tor-selected portion of the image through particular 
movements, such as pan, zoom, up/down, tilt, rotation, 
etc. 

By emphasizing the peripheral content of a scene, 
the visual imaging system can use a single camera to 

20 capture the relevant visual information within a panoram- 
ic field of view existing along the horizon, while being 
able to conventionally store and easily display the scene, 
or porttons thereof, in real-time. Using a single optical 
system and camera is not only cost-effective but keeps 

2S all hemispheric visual data automatically time-synchro- 
nized. 

One advantage of the present invention is that the 
unique visual imaging system lens can capture informa- 
tion from a hemispheric scene by emphasizing the pe- 
so ripheral portion of the hemispheric field of view and thus 
provide greater resolution with existing imaging devices 
for the relevant visual information in the scene. As an 
example, if an ordinary fish-eye lens focuses the lowest 
1 5 degrees up from the horizon on 10% of the imager at 
35 the objective plane and the peripheral-enhancing lens fo- 
cuses that same 1 5 degrees on 50% of the imager, there 
is a 5-fold increase in resolution using the same imaging 
device. Depending on the application and exact formu- 
lation of the lens equations, there will be at least a 5X 
40 increase in resolving power by thrs lens/imager combi- 
nation. 

Another advantage of the present invention is that 
the captured scene information having an enhanced pe- 
ripheral content can be stored as a single image using 

4S conventional storage techniques, allowing post-acquisi- 
tion selection of particular views. 

Still another advantage of the present invention is 
that the image can be read out ^rom storage and elec- 
tronrcally manipulated at any time after acquisition to pro- 

so vide flexibility in display options, as opposed to being 
stored in conventional perspective formats only. 

Another advantage is that the system can generate 
motion video from a reduced set of still images through 
interpolation and inter-scene warping. Still other advan- 

55 tages of the present invention should become apparent 
from the following detailed description and appended 
claims. 

The following detailed description is made with ret- 
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erence to the accompanying Figures, wherein like refer- 
ence numerals refer to like parts. 

In the annexed drawings: 

Figure 1 is a schematic illustration of the various 
components and organization of the visual imaging 
system of the present invention; 

Figure 2A is a cross-sectbnal diagram of a wide 
angle lens system of the prior art; 

Figure 2B is a box diagram indicating the field input 
and output rays and the resulting relative field cov- 
erage a lens of the prior art typically provides in an 
image plane for detection by an imager device; 

Figure 2C is a cross-sectional diagram of a lens sys- 
tem of the prior art indicating angular key to visual 
field; 

Figure 3A is a cross-sectional diagram of one 
embodiment of a peripheral image enhancing lens 
system of the present invention; 

Figure 3B is a box diagram of the annulus field input 
of a bundled fibre array portion of the lens system of 
Figure 3 A; 

Figure 3C is a box diagram of the imager interface 
output of a bundled fibre array portion of the lens 
system of Figure 3A; 

Figure 4A is a cross-sectional diagram of another 
embodiment of a peripheral image enhancing lens 
system of the present invention; 

Figure 4B is a box diagram indicating the field input 
and output rays indicating the resulting field cover- 
age of the lens system of Figure 4A; 

Figure 5 is a schematic representation of the map- 
ping locations of the lens systems of the present 
invention resulting upon an imaging device; 

Figure 6 is a schematic block diagram of the image 
processing circuitry of the present invention includ- 
ing the transform processor subsystem thereof; 

Figure 7 is a block diagram schematk:ally indicating 
further integration of a portion of the image process- 
ing circuitry of Figure 6, and 

Figure 8 is an alternate emlxxltment of the image 
processing circuitry of the present invention incor- 
porating the integrated circuitry elements of Figure 
7. 
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The invention will be defined initially with a -brief <ie- 
scription of the principles thereof. 

The present inventkMi stems from the realization by 
the inventors that in many of the technical hemispheric 
5 field applications, where the image detector is parallel to 
the horizon, much of the relevant visual information in 
the scene <eg trees, mountains, people, etc.) is found 
only in a small angle with respect to the horizon. Although 
the length of the arc from the horizon -containing the rel- 
10 evant information varies depending upon the partcular 
application, the inventors have determined that in many 
situations, alrTK>st all the relevant visual information is 
contained within about 10 to 45 degrees with respect to 
the horizon. 

IS To maximize datacollectron and resolutk)n lor anal- 
ysis and/or display of the relevant visual informatksn lo- 
cated in this portion of the hemispheric scene, it is desir- 
able to maximize the dedication of the avai<able image 
detection area to this peripheral field portion. To accom- 

20 nrKXlate this, it is necessary that the •central" portion of 
the scene (from 45 to 90 degrees with respect to the ho- 
rizon) cover only the renr^ining ar-eas of the imager plane 
so as not to intertere with light from theperiphery. 

In many cases, since the -central" areacontair^ less 

2S detailed information, such as a solid white ceiling or a 
clear or lightly-clouded sky, it is allowable to maximize 
completely the dedkation of the available image -detec- 
tion area to the peripheral field portion by reducing the 
portion of the imager <Jevk:e representing the Central' 

30 area to near zero. Of -course, in certain instances, it is 
desirable «to analyze this less detailed information, -but 
this portk>n of the scene canhe m inimized to some extent 
without significant degradation of such visual informa- 
tion. As will be described herein In more detail, the 

35 present Inventron provides two manners^Example I and 
Example II) for capturing, storing, and selectively dis- 
playing the critical visual information in a scene for many 
important applications. 

Referring now to the drawings, and initially to Figure 

40 1 , the visual imaging system of the invention irx^tudes a 
still image or moving pcture camera 10. having a tens; 
indicated generally at 14, designed 1o capture and en- 
hance the peripheral content of a hemisphere scene. 
The captured scene can be stored onto an assortment 

46 of media, eg photographic film 16, electronic storage 18, 
or other conventional storage means. -El ectronk; storage 
18 is preferr-ed because of the ease of electrons manip- 
ulation thereof. AdditkDnally. photographic film 16 re- 
quires an Image -scanner 20 or other capture-andcon- 

so verston method to change the image into electronk; for- 
mat before electronic manipulatroncan be perfomned. 

The stored electronic innagexlata is then selectively 
accessed by a transform processor engine -22 artdcan 
be electronk:aliy manipulated aocofding to user-defined 

55 criteria, such as pan, up/down, zoom, etc. The transform 
processor 22 corrects the image for display on a conven- 
tional display device 28 in a normal viewer format or on 
head-mounted displays 30, in whk:h integrated orienta- 
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lion-sensingdevices+iaving. for example, wearer eye fo- 
cus detection function, can be used to manipulate and 
define the user controls. 

I Image Acquisition 

a. Camera 

The camera 10 for the visual imaging system is an 
optical device that is capable of receiving a focused im- 
age from a lens and transforming that image into an elec- 
tronic signal or into hard copy storage such as photo- 
araphic film. Various types of cameras for wide-angle 
viewing are l^nown to those in the art. such as 35mm 
cameras. 8mm cameras. NTSC. RSI 70 and HDTV-type 
cameras. The present invention is designed to be com- 
patible with most commercially-available two^limen- 
sional carr>eras. with scaling ot the lens geometries. It 
also has the technological capability to be applied to 
three^iimensional cameras. The<:amera can be mount- 
ed and supported in a conventional manner. 

^ ■ wiih Enhanced Peripheral Content 



The fundamental principle behind the enhanced pe- 
ripheral content lens is the selective magnification of the 
periphery and the focusing of more of that content on the 
objective plane. This recognizes the current limitations 
of imaging devices and film with regard to resolution. As 
such, the more of the peripheral content that can be fo- 
cused on the objective plane surface, the more points of 
data that can be resolved with a given density of imaging 
device or material. Therefore, for this new class of se- 
lective magnification lenses, the surface area of the im- 
aaing plane resen/ed for peripheral content will be large 
relative to the central content and roughly similar for all 
lenses in this class, regardless of whether the lens ,s de- 
siqned for 1-10 degree peripheral emphasis or 1-45 de- 
qree peripheral emphasis. However, it should be noted 
That the lens with 1-10 degree emphasis will have much 
better resolution for the same objects than the lens with 
1-45 degree emphasis. 

The lens 14 for camera 10 provides a collection oT 
data for enhanced digital processing of the peripheral 
portion of a given field of view. The lens uniquely 
achieves this by filling the greater available area of an 
imager device with the peripheral areas rather than the 
central areas of the captured scene. 

A periphery-enhancing optical system suitable for 
achieving thesis of the present invention can be con- 
figured in various ways to present an inrege of the field 
free of detrimental values of critical aberrations. Two ex- 
amples or embodiments are preferred. Example l^s il- 
lustrated by figures 3 A-3C, preferably is a multi-medium 
system comprising a wide-angle multi-element optical 
lens and a fibre optic imager device. Exampfe II as illus- 
trated by Figures 4A-48. preferably is acombination sys- 
tem comprised of multiple refractive optical elements, 



one of which is of gradient index material and hemispher- 
ical in shape. The inventors define that other configura- 
tions relying on binary optics could also be utilized to ac- 
complish similar results. 

5 The system of Example I is best suited for applica- 
tions where the capture of the minimal information con- 
tained in the central area is irrelevant A coordinated fibre 
bundle array 40. either rigid or flexible, carefully aligned 
and assembled by methods known to those skilled in the 

70 art is used to capture a peripheral annulus ot the wide 
field image provided by the multi-element wide field lens. 
The multi-element wide fieW lens 42 therefore, although 
covering the same field as standard wide field tenses, is 
required to provide a larger image of the entire field. This 

IS can be accomplished by those skilled in the art by scahng 
a standard lens to a larger focal length until the desired 
image size is reached. (REF. Journal of theOptical So- 
ciety of America. 1964, Lens Designs Presented by Mi- 
vamota; Smith, W.J.. Modem Lens Design. Ch. 5.4 

20 -scaling a Design". McGraw-Hill. Inc. 1992. Also see 
U.S. Patent No. 4.266.373 by M. Horimoto for additional 

wide-field lenses.) 

The fibres, arranged in a specific mechanical man- 
ner deliver the information in a geometrically annular 
25 shape from the peripheral image area to either the rec- 
tangular or circular shape of a specific imaging <Jev»ce. 
Specific configurations can be devised for desired pe- 
ripheralcoverage (up to 10 degrees through 45<tegr«es) 
and matched to required image detector plane geometry. 
30 (REF Sect 13. Fibre Optics. Handbook of Optics, Mc- 
Graw-Hill Inc.. 1978.) Referring to Figure 3A. for a de- 
sired angle 0 (theta), a standard wide-field lens design 
is scaled in -f". the focal length, to provide ry' such that 
the minimum pick-up area of the camera imaging devK^e 
35 <2r') is met. In the "Camera/lmager Interface' portion of 
the figure, the equation for fibre bundle radius is related 
to the imager is: 



40 



Area at innager = {pi)ry.^ - {pi)^.^ = (P0( ' ^x- ) 

2 2 

Imager length = 2r'. where r = r^ - r^. 



A configuration in the form of Example I achieves an 
image in which the peripheral portion (0-45 degrees with 
respect to the horizon) of the field of view preferably en- 

45 compasses between approxinnately 90% to 100% of the 
entire usable area of the imager device, rather than the 
35% or less that is typical with commercially-available 
wide-angle, fish-eye, or other conventional lenses. A 
configuration in the form of Example II achieves an im- 

60 age in which the peripheral portion preferably enoom- 
passes between 50% to 70% of the usable area of the 
imaging device surface. 

The system of Exanr^ple II is best suited for applica- 
tions where all information contained in the field of view 

55 is relevant, although to differing <iegrees. The combina- 
tion system of Example II relies on the principle use of a 
Bravais System where a hemispherk:al lens 52 or a com- 
bination of lenslets magnify an object to an image, 
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wherein the object and image lie in the same plane. In 
this case, the hemispherical lens 52 Is nnade of gradient 
index material such that points impinging the image 
plane incur different factors of magnification depending 
on what index of refraction and what portion of the hem- 
ispherical curve or lenslets they pass through. This con- 
cept is used in combination with a mufti-element refrac- 
tive lens 54 that is designed to capture a wide field of 
view and also compensate for the colour aberration In- 
duced by insertion of the hemispherical lens. This colour 
compensation can be designed by those skilled in the art 
by using a computer optimization routine, (REF. M. Hori- 
motoand U.S. Patent No. 4,256.373.) By use of the hem- 
ispherical gradient index unit with a wide field multi-ele- 
ment lens, the portion of the camera dedicated to the pe- 
riphery is increased, thereby Increasing the relative res- 
olution of information detectable by the imaging device 
sensing elements. With the gradient index hemisphere, 
index values decrease from the centre, such that: 

n^ < ng < ng < n^ 

The operation of the optical systems is schematical- 
ly illustrated in Figures 2A-B, 4A and 48. In Figure 2A. 
arc X represents the field of view of the "centre" of the 
lens system, while the arc Y represents the practically 
usable portion ol the "peripheral" field. The areas X' and 
Y' in Figure 2A represent the resultant image focal loca- 
tions on the imager at the object plane. Arc Z and area 
Z' represent areas outside the nonmal Imaging range of 
the lens. (No actual demarcation lines would exist in ac- 
tual use; this is merely for explanatory purposes.) 

Figure 2A represents a typical wide-angle type lens 
32. while Figure 4A represents a lens constructed ac- 
cording to the principles of the present invention. As 
should be apparent from comparing Figures 2 A and 4A, 
a typical wide-angle type lens has a fairly significant por- 
tion of the image surface dedicated to the central field of 
the lens; while the lens constructed according to the 
present invention has a fairly significant portion of the 
objective surface dedicated to the peripheral field - and 
consequently less of the surface dedicated to the central 
field. 

The portion of the Image surface used for the periph- 
eral portion of the scene (as compared with the central 
portion of the scene) can vary depending upon the par- 
ticular prescription of lens specified, which is selected to 
capture items of interest for a given application. For ex- 
ample, if the imaging system is used to capture a pano- 
rama of an outdoor scene, the relevant visual information 
may be contained within 10 degrees of the horizon. The 
lens of the present invention can thus be designed to en- 
hance only the field of view within 10 degrees of the ho- 
rizon. On the other hand. If the imaging system Is being 
used to capture a room scene within a building, the rel- 
evant visual information may include objects on wails 
and thus be contained within about 45 degrees from the 
horizon. A peripheral enhancing lens can thus also be 
designed to enhance the field of view up to 45 degrees 



from the horizon. Of-course. the enhanced portion of the 
field of view depends upon the needs of the particular 
application, with the enhanced portion preferably falling 
somewhere between these two extrenrtes. In anyose. 
5 the principles of the present invention can be applied to 
these types of situations with equal success, wKh any 
appropriate corrections being made with the image 
transform processors, as will be described tierein in 
more detail. 

10 As illustrated In Figure 3A, a preferred form of the 
Example I form of optical system comprises a standard 
wide field lens 42 and d'coordinated fibre array 40. The 
focal length of the wide field lens is scaled in order to 
match the peripheral field of view desired. Thevcooidi- 

is nated fibre array-consists of an annular input face 44 that 
collects the Image projected from the standard wide field 
lens. The fibre bundle array then redirects the informa- 
tion from the peripheral view to its output end by total 
internal reflection. Fibres capable of 3 micron accuracy, 

20 covering the area of the annular region, are coopdinated 
into a rectangular -or circular shape at theiroutput. -de- 
pending on the geometry of the^orresponding imaging 
.device. The size of output is also matched to the camera 
imaging device used. 

2S Of course, these system-configurations and param- 
eters are only exemplary in nature and other configura- 
tions and parameters of the present Invention could be 
used to provide enhanced peripheral imaging capabili- 
ties, as should be apparent to those skilled in the art. 

30 

a. imaoer Oevice 

An electroniccamera 1 0 used for exemplary purpos- 
es Includes an imager device that records the optical im- 

35 age from the lens at the object plane. f=or a photographic 
process, the imaging medium is film, while for an elec- 
tronic process, the imaging medium is an electronic-de- 
vice such as a charge-coupled device «(CCO) or 
charge-injected device <C10). As iridjcated previously. 

40 electronic processes are typically preferred over photo- 
graphic processes as they are easier to electronically 
manipulate. Photographic processes, however, can be 
preferred in certain situations. Numerous makes and 
models of film and electronic imaging devices are known 

45 to those skilled In the art that provide uniform resolution 
across the receiving surface. 

When the Imager device is used with a lens con- 
structed according to the principles of the present inven- 
tion, the imager <iev.ice will coHect more informatk)n 

so along the horizon than it wilt from the central area of the 
hemtspherk: scene. With only a given limited resolution 
of either film emulsions or CCD pixel density, the lens 
focuses more useful information at the objective plane. 
The peripheral portkjn of the scene will therefore have a 

ss higher relative resolution in the resultant transforrr)ed im- 
age than thecentral portbn. Thus, the details of any ob- 
jects along the horizon will be highly aocentuated. fur- 
ther, any distortion {3g spherical aberration) thattxx:urs 
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in the peripheral region of the lens will be Imaged onto a 
arger surf ace and can thus be more easily and f ully com- 
pensatedfor. The Image mapped onto the imager dev«e 
can be described by a series of concentre circles, as 
schematically illustrated in Figure 5. As an exampte 
each circle a. b, c, d. etc. on the ^'^^'J^y^'fl!'^ 
described by radii of arbitrary unrts. e.g.. 2. 6. 9. 14, eic., 
respectively The radii of the circles depends on the mag- 
nification of the different regions of the hemisphere, with 
the sections of the outer circles having a greater area as 
the magnification of the peripheral region increases. For 
example, in an arbitrarily selected illustration case, each 
concentric circle represents 1 8 degrees field of view from 
the horizontal plane. w»h the outer circumference of the 
outermost circle being level with the horizon. The inven- 
tors havedetermmedthat the arc subtending the twoout- 

er circles (I.e.. 36 degrees from the horizon) contains the 
relevant infomation in many hemispheric scen^ for 
many applications (although this value can be varied de- 
oending upon the particular application). 

Cateulating the total circular area of the entire image 
circle yields 1257 units squared. The area of the three 
inner circles is 254 units squared. Therefore, the two out- 
er circles contain about 80% of the usable area on the 
Imaging device. Note that the image blocks correspond- 
ing to the horizon are spread across more area on the 
imager devtoe than those in thecentralareaof the irnage. 

Thus the image blocks of the imager device are domi- 
nate<i by objects along the horizon, and those are the 
area of interest. This correlates to greater resolution for 
the peripheral areas of the scene. 

If for some reason an Example I circular-output op- 
tteal system is configured with a square or rectangular 
imaging device, the comer areas of the Imaging device 

nci useful for photographic purposes. However, 
these areas can be used to store other lnformat«n. for 
example, such as digitized audio <^'^'''°'°^^X 
ondaiy documentation from the scene, if the »T«ger is 
radially mapped 1:1 with electronic memory. 

11 Image Storage 



The image received on the Imager device is passed 
on to the system components for storage. For pho^c^ 
graphic processes, a storage device might be film; wh^e 
tor electronic processes, the storage devK:e might be 
electronic storage in the form of random access memo- 
ries, a conventional diskette or hard file, or ydeo record- 
ing tape. The entire display of the scene (along with any 
secondary documentation) can be stored as a single im- 
age on the storage device. 

The image is stored in a "warped' form. The wa^ed 
image is caused not only by the wide-angle nature of me 
lens (i.e. the -keystoning" effect), but also by the en- 
hanced peripheral field of view of the lens (i.e.. magnifi- 
cation a^g the periphery). The ""^^'y'"9 J°"2!P','^ 
that a partial sliceof the scene can be reproduced with 
the proper aspect ratio for the human visual system (i.e.. 



as a oerspective corrected view). 

M schematically diagrammed in Figure 6, the stored 

imaoe is loaded into the source image buffer 40 if it has 
been stored in electronic fomi on a host system such as 

5 a personal computer or controller. Alternativehr. the im- 
aqe can be brought in for processing without going 
through storage. As one option, an analog signal from a 
Video camera can connect Into an NTSC-to^igital con- 
vener«0 This converts the image from analog informa- 

10 tion into a digital bit map (i.e., into "pixels"). The source 
Image is then loaded into the source image frame buffer 
62 However, as indicated previously, any type of camera 
can be used to provide the electronic input to buffer 62. 
The bufler preferably operates with sufficient speed so 

IS that real-time viewing is possible. 

Ill Image Retrieval/Display 

The stored image can be selectively accessed and 
20 transformed for display. If the storage Is photographic 
film, an image scanner may toe used toconvert the stored 
imaqe into an electronic format for subsequent manipu- 
latiai In order to recreate a proper display of the scene 
in two dimensions for perspective correct viewing, proc- 
zs essor logic In transfomi processor engine 22 is utilized^ 
The transform processors may be made of collections o1 
small-scale, medium-scale, large-scale. or 
very-large-scale Integrated (VLSI) circuits, examples of 
which are Image resampling sequencers such as frie 
30 TMC2301 and TMC2302, commercially available from 
Raytheon Semiconductors (formerly TRW LSI Products, 

Inc., LaJolla, CA.). 

In Figure 6, resampling sequencers control the ad- 
dress sequencing of the pixels in the source image tjuffer 
3S 62 through a multiply/accumulate unrt 64. and from there 
into the warped image buffer66. The sequencers control 
the filtering or remapping of 2-dimensional images from 
a set of cartesian cooidinates (x.y) as defined within 
each sector "unit" (A1 , B1 , etc.) onto a newly transfomied 
^ set of coordinates (u,v). The -fish-eye" type of transfor- 
mations described in U.S. Patent No. 5.185.667 are 
based on non-constant second-order derivatives. A dif- 
ferent set of second-order derivatives would be em- 
oioved for the transfomis associated with the peripher- 
4S al-enhancing configurations of the present invention. 
The sequencers can also handle three-dimensional im- 
ages by resampling them from a set of Cartesiari coor- 
dinates (x.y.z) Into a new. transfomied set (u.v.w)^Typi- 
cally these sequencers can support nearest-neighbour. 
so bilinear interpolation or convolution resampling, andean 
operate at speeds allowing real-time operation. 

Remapped pixel locations (i.e., interpolation ker- 
nels") of more than one pixel in the bit map ^n 
external interpolation coefficient look-up table€8 andt»>e 
ss multiply/accumulate unit 64. A table "walk" is typically 
perfomied on each source pixel, thus providing a 
smoother image by summing the products of the original 
lens image data with the appropriate interpolation ooef- 
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ficients. By capturing the hemispheric lens image data 
into source innage buffer 62. the warp engine can be pro- 
grammed to perform a perspective correction, much like 
an inverse keystoning effect. The remapping of the pixel 
locations is matched to the differential magnification of 
the particular periphery-enhancing lens system used. 

Direct access to the interpolation coefficient look-up 
table 68 and to the transformation parameters is also de- 
sirable to allow dynamic modification of the interpolation 
algorithm. Thus, a local interpolation coefficient buffer 70 
to update the varying transform parameters is included 
to allow for real-time still and motion innage transforma- 
tions. 

The row and column warping engines 72a, 72b of 
the transform processor 22 supply addresses to the 
source image buffer 62. The addresses are determined 
by the interpolation algorithm chosen. The multipty/ac- 
cumulate unit 64 takes the pixels supplied by the source 
image buffer 62 under warping engine control and mul- 
tiplies the pixels together using combinational togic with 
weighting factors dependent on the algorithm. Compen- 
sation for aberratton (e.g., spherical aberration) can also 
be made at this point; Finally, the composed interpolated 
pixels are sent to the warped image buffer 66. The ad- 
dress location within the warped image buffer is again 
determined by the warping. The algorithm parameters 
from look-up table 68 are input to the registers of the row 
and column warping engines 72a, 72b, as well as into 
the interpolation coefficient buffer 70. 

The memory controller/clock circuitry 74 provides 
refresh control to the source and warped image buffers 
62. 66. In addition, all clock sources are synchronized 
through this circuitry. The bus interface and control cir- 
cuitry 76 also provide an interface to the host system bus 
(ie for MCA, ISA, etc.) and the remapping circuitry. This 
interlace k>gic sen/es to load control information into the 
remapping circuitry and to provide a path to transport 
warped images to the system display buffer (not shown; 
part of host system), or store innages to disk via the sys- 
tem bus prior to warping. An optbnal random access 
memory digital-to-analog converter (RAMDAC) 78 pro- 
vides support for a local display connection if desired. 

One feature of the transform processors is the valid 
source address flag within the transform processor en- 
gine. This allows the user to construct abutting subimag- 
es in the (x.y) plane without danger of edge interterence. 
Thus, edge detection of the unused areas outside the 
circular image of Figure 5 can alert the system to ignore 

these values. 

The image capture function can be accomplished 
with either still or motion video devices or as pre-record- 
ed digital data. All types of image data are input to the 
source image buffer 62 for processing as desired. Note 
that while the preferred mode for still image capture will 
come from previously captured images through a local 
host bus interface 76, the NTSC digitizer 60, for example, 
can provide real-time data from an external video cam- 
era. Any similar device that converts an image to the ap- 



propriate digital format for input to the-souice image buff- 
er can be substituted (or<iigitizer'60. In the same fashion, 
an electronic still imager, such as an electronic still<:am- 
era, line scanner, or table scanner, 'can provide still im- 
5 age data for processing. Prerecorded distorted images 
generated through this invention's optical system -can 
also be input through an additbnal -conversion devk:e4o 
allow dynamk: manipulation of pravk>usty recorded im- 
age data. 

10 The image transformation pBrforrT>ed on the cap- 
tured digitized image from modified hemispheric coordi- 
nates to planar coordinates for display is one of a multi- 
tude of possible image transformatk>ns, any of whichtcan 
be invoked in real-time for smooth merging of eftacts. 

IS These tran6formatk>ns include, 'but are not limited to 
pans, up/downs, zooms, tilts, rotations, scalir^, cropping 
and image shear, which can be controlled using human 
or computer input. Image filtering can be performed as 
well as edge detection in associated processes during 

20 the course of manipulation. These -services -can be ap- 
plied to any system image k^aded into the source image 
buffer, thus providing a host of added features beyorui 
the simple application of the hemispheric lens and dis- 
play system. 

26 The advantage of the image transformatran logic be- 
comes apparent when describing partrcular appi cations. 
A security camera application can be implemented to 
view an entire panorama such that the security monitor 
will display full-nriotion rate images in real-time. Source 

30 image coordinates for still images can also be se- 
quenced, allowing perceived animation or full-rTK>tion 
renditk>ns by simply reloading new source image-coor- 
dinates into the warping engines as frames are drawn 
from memory. Details from other stored irnagescan be 

35 utilized to give the effect of f ull-motran panning of the ho- 
rizon within the captured image by the lens. 

An addittonal feature of the present invention is its 
ability to create motnn vkJeo presentations with a re- 
duced set of still frames, as compared with the numtDer 

40 of full-motion frames ordinarily required. 'For «xanr>ple, 
when a set of buikJtng storefronts is filmed with a tradi- 
tional motron picturecamera from the back of a moving 
truck, each frame contains the tirr>e-basedcorrtent of the 
image entering the traditional iens and only one limited 

45 field of view is available at a time. With the system of the 
present invention, nrx>tioncan be reconstructed from a 
greatly /educed subset of frames because the greater 
field of captured data already contains picture content 
from the next frame andprevk>us frame due<to its excep- 

so tionally wide angle. By analyzing pcture content within 
a host computer program, intermediate frame ^uiva- 
lents sufficient to complete a full-nnotion data set can be 
constructed and fed to the soufce image buffer in se- 
quence for processing. Alternativety, separate circuits 

55 operating at real-time rates can interpolate intermediate 
values and supply changes in transformaton parame- 
ters for existing still innages fast erwugh to synthesijEe 
motion sequences. Prelerence for one technique over 
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another will be dependent on the specific application re- 
quirements and other development/market considera- 
tions. 

Finally, the transform processor -subsystem can pro- 
duce multiple different outputs simultaneously from indi- 
vidual stored or currently-converted images. With the 
main transform processor circuits collected into a simpli- 
fied single image processing subsystem 80 as shown in 
Figure 7, multiple outputs may be generated from a sin- 
gle image source, either motion or still, with individual 
effects for each scene as desired, altowing several 
scenes on different display devices or several windows 
on a single display. This is accomplished by incorporat- 
ing several image processing subsystems 80 within one 
overall system, as shown in Figure 8. 

in all cases, by having greater resolution of the pe- 
ripheral image of a scene, the details of any objects along 
the horizon will be enhanced. Further, aberrations occur- 
ring around the periphery of the lens (i.e.. spherical ab- 
errations) can be more fully and completely compensat- 
ed for. as the aberrations are spread across a greater 
area on the imager device. 

As described above, the present invention provides 
a visual imaging system that efficiently captures, stores, 
and displays visual information about an enhanced hem- 
ispheric field of view existing particularly along the hori- 
zon, and that allows electronic manipulation and selec- 
tive display thereof even after acquisition and storage, 
while minimizing distortion. 

The principles, embodiments, and modes of opera- 
tion of the present invention have been described in the 
foregoing specification. The invention that is intended to 
be protected herein should not, however, be construed 
to the particular form described as it is to be regarded as 
illustrative rather than restrictive. Variations and changes 
may be made by those skilled in the art without departing 
from the spirit of the present invention. Accordingly, the 
foregoing detailed description should be exemplary in 
nature and not limiting as to the scope of the invention 
as set forth in the appended claims. 



Claims 

1 . A system for electronic imaging and manipulation of 
a hemispheric field of view, comprising: 

a camera for receiving optical images of a 
hemispheric field of view and for producing output 
signals or affecting pliotographic film-based materi- 
als corresponding to the optical images; 

an optical imaging device associated with said 
camera for producing optical images throughout the 
hemispheric field of view for optical conveyance to 
said camera, said optical system having a configu- 
ration adapted to capture and enhance an image of 
peripheral regk>ns of the hemispheric f lekJ of view; 

an image processing device associated with 
said camera and sakJ optKal innaging system for 



receiving optical images from said lens and for pro- 
viding digitized output signals representative of the 
received optteal images; 

input image memory for receiving the digitized 
5 output signals f rom said imager device and for stor- 
ing the digitized output signals; 

image transform processing circuitry for selec- 
tively accessing and processing the digitized output 
signals from said input image memory according to 
70 user defined criteria; 

output image menrrary for receiving the proc- 
essed signals from the image transform processor, 
and 

an output display or recording devk:e con- 
15 nected to said output image memory for recording 
the signals in said output image memory. 

2. The system as claimed in claim 1 , wherein sakJ opti- 
cal system magnifies a portbn of the field of view 

20 within a range of approximately zero to forty-five 
degrees above the horizon of a captured image of a 
hemispheric fiekl of view. 

3. The system as claimed in claim 1 or 2. wherein said 
25 optical system has a configuration which images a 

peripheral portion of the hemispheric scene onto at 
least 50% of an imaging area of an imaging device. 

4. The system as claimed in claim 3. wherein said opti- 
30 cal system includes a wide field multi-element lens 

positioned to direct an image to a light transmitting 
fibre array. 

5. The system as claimed in claim 4 wherein the fibre 
35 array is geometrically arranged to have a generally 

annular input end and a generally rectangular output 

end. 

6. The system as claimed in claim 4, wherein the fibre 
40 array is geometrically arranged to have a generally 

annular input end and a generally circular output 
end. 

7. The system as claimed in claim 4 wherein the fibres 
45 of the fibre array have an imaging accuracy on the 

order of three microns. 

8. Tbe system as claimed in claim 4 wherein the focal 
length of the wide field lens is seated to match a 

so desired magnification of the peripheral field of view. 

9. The system as claimed in any one of claims 1 to 8, 
wherein said optical imaging system includes a col- 
our-aberrated multiple element wide field lens in 

55 combination with a gradient index hemisphere lens. 

10. A method for electronically capturing, storing, and 
manipulating a hemispherk: fiekJ of view, comprising 
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the steps of: 

providing an optical system having a configu- 
ration that enhances the peripheral portion of the 
field of view. 

capturing the hemispheric field of view with the s 
periphery-enhancing optical system and imaging 
the field of view onto an imager device by enhancing 
the peripheral field of view, 

storing the captured image as a single image, 

selectively accessing a ponion of the stored io 
image according to user-defined criteria. 

transforming the stored image so that the 
stored image can be displayed as a perspective-cor- 
rect image. 

displaying the perspective-correct image in a 
. user-defined format. 



20 



25 



30 



35 



40 



45 



SO 



55 




14 



EP 0 695 085 A1 



SO 




Angular Key to 
Visual Field 




•:::f.:-7r. 











* / f 




'ft 

* I I 

* / / 


\ \ * 
I \ * 


* / / 
/ / / 

mm u 
* M t 


\ \ * 
\ \ * 
\ \ * 


9 M i 
* J / 

* / / 
' # / 


\ \ * 
1 \ • 

I \ * 
\ \ * 
\ \ * 



Border of imager 



2' 




View €8' 



FIG. 2 



15 



EP 0 695 085 A1 




Border of imager 

Note: Artist's rendition FIG. 3 A 

Drawing not to scale - 



1« 



i 

EP 0 695 085 A1 




Note: Artisf s rendering Border of imager 

Drawing rK>t to scale 

FIG. 3B 



17 



0 



EP 0 695 085 A1 




18 



£P 0 «95 085 A1 



Image dota from electronic Imager 



KfTSC or other enalog 
Input from comera 



Panoramic Image 
Trenstorm. Pfirametors 

Lookup Tflbie 



Host 



Bus 



E>eta 



Control 



Address 



Bus Interface/ 
Control 



7^ 



7^ 



L_L 



Memory 
Control 



Analog to ' 
-OigHol 

Conversion 



Row<X) 
Warp 
Engine 



Source Address 



/ Address 



Interpolation 
Coefficient 
Butter 



^ ^ Address 



Column (Y) 
Warp 
Engine 




iCoeff. Data 



Multiply/ 
Accumulate 

Unit 



Warped 



Destination Address 




/ 



Warped 
V OataOut 



/ - Multiple lines, such as at>us. 




To Deploy 



FIG. 5 




EP 0 695 085 A1 



Panorcimic Image 
Transform. Peirametors 
Lookup Tablfl 



NTSC or other analog j 

Input from camera ^ J 

; Analog to 
: Digital 
I Convarvion 



Image data Irom 
electronic Imager 




Bus Interlace/ 
Control 



.-J 



To Otsptay 



To Display 



To Display 




Source Address 



nterpolation 
Coefficient 
Buffer 



7^ 



Address 



Column (Y) 
Waip 
Engine 



7^ 



Coelf. Data 



r 



Source 
Image 
Buffer 



Source 
""Oata 



Multiply/ 
Aocumulflte 

Unit 



Destination Address 




FIG. 6 



is shown above as 



Image 
Proceaaing 
and 
Warping 
Orcutts 



/ = Multiple lines, 
such as a bus 



20 



EP 0 695 085 A1 




Curopcao Patent 
Office 



EUROPEAN SEARCH REPORT 



EP 95 30 4930 



DOCUMENTS CONSIDERED TO BE RELEVANT 




Calcfory 


Citation «f dociunent wilh inJictlon, where appropfiatVt 
of relevant passages 


Rcfevaat 
lo claim ^ 


CLASSIFICATION OF THE 
APPIJCATION 0B1.CL*) 


D.Y 


WO-A-92 21208 (TELEROBOTICS INTERNATIONAL, 
INC.) 

* page 1, line 1 - page 9, line 25 * 


1.10 


H04N5/262 


V 
T 

• 


WU A 07 UoU/O ^u.lS. otUKLIAKT rUK iKAUt 6i •* 

INDUSTRY) 

* page 4, line 10 - line 16 * 






A 


PROCEEDINGS OF THE SPIE. APPLICATIONS OF J 

UlGi lAL iflAbk rKULtbblNb XV, 

vol. 1771. July 1992 SAN DIEGO, CA, USA. 
pages 106-112, 

GILBERT ET AL. * Endoscopic inspection and 
measurement' 

• page 106. line 24 - page 108, line 10 * 


1-3,10 




< 


PROCEEDINGS OF THE SPIE, APPLICATIONS OF , 
DIGITAL IMAGE PROCESSING XV, 
vol. 1771, July 1992 SAN DIEGO, CA, USA, 
pages 567-573, 


1-3,10 






•GREGUSS ET AL. *PALINADAR: a PAL-optic 
based Imaging raodul for allround data 
acquisition and recording' 
* page 567, line 17 - page 569, line 43 * 




TECHNICAL FIELDS 
SEARCHED (lisf.CLA) 






nufw. 
€028 


a 


EP-A-O 574 325 (THOMSON-CSF) 

* column 1, line 1 - line 18 * 

* column 3, line 35 - line 45 • 

* column 4, line 38 - column 5, line 33 * 


1.4.5.K 


>< 


m 

A 

i 


EP-A-O 547 635 (EASTMAN KODAK COMPANY) 

* column 3, line 4 - line 18 * 

* column 3, line 46 * column 4, line 2 * 

* column 5, line 3 - line 33 * 

* column 6, line 39 - line 44 * 


1.10 




A 


US-A-4 832 472 (ROBB) 

* column 2, line 45 - line 53 * 


9 




Tbc present search report bas beco 4lr«wQ op fpr aU dauns 


* 


1 



THE HAGUE 



IM» *m««Mm alUv I 

27 October 1995 



Oe Dieuleveult, A 



CATEGORY or CITED DOCUMENTS 



X : pafticttfauiy idcvaal U takm aloot 

Y : parttcnUily idcvaai if coaiUacd with anotlii 

tocniBcnt of the sane catcgoiy 
A : fcc to o to glcil bmdkt^wxni 
O : BOD-wrhta tfisdontc 
F : ipt uw ii if dacaacal 



T : tboDty or pnsdjric en ic rty tag lb« ia«iBliim 
E : t%Ma patent *otx»naa, bot paUisbcd oa, m 

«ft« ckc fBlat tett 
D : tecoHit dt«i la ite •ppUcadoa 
L : 4ocaaMil died tar < 



a : mtfafcv of ibt 



pattai taaily, comspoadlDg 



21 



EP0«95 085 A1 




EuropciLO Patent 



EUROPEAN SEARCH REPORT 



AppHcsiin Nnmbcr 

EP 9S 30 4930 



DOCUMENTS CONSIDERED TO BE RELEVANT 



PATENT ABSTRACTS OF JAPAN 
vol. 13 no. 425 <M-873) ,21 September 1989 
& JP-A-01 163079 (NIPPON SHEET €USS CC 
LTD) 27 June 1989 « 
* abstract * 



Ckadoa of dvcumcnC witfa 
ff rdevwitp: 



rawn for lil di 



Rdevant 
ttt dann 



OSSIFICATION OF TIIE 
APPLICATION (lB<-CLfi) 



9 



TECHNICAL FIEUDS 
SEARCHED (Int.CL() 



8 

2 

m 

3 
i 

i 

m 

c 

o 



rtmxmlwtmtk 

THE HAGUE 



mttkemn 



27 October 1995 



Oe Dieuleveult, A 



CATCCORY OF CITED DOCUMENTS 



X : particxlafty rdevaat if ta^cn slose 
Y : pMtlCBlafty rdevmat if c oi at M iri witb 

iocnicnt of Ike sum catcgoiy 
A : (•ctoolocical badienMSd 
O : tuMk minm iisctesorc 
P : iRtcraciiite iecucnt 



T : ikeoiy or pciadplc ■■rferlyiag th« tovcatloo 
E : oillv potent 4ocu— f, bin poUiskei oa, mt 

after Ae fUtac 
D : #ocnai«ii dt«i ta tke appllcatfoa 
L: iocaaMBt otcd for othci 



A : moabcr of tite suae paical favibri cetrospoadinc 
4ecni 



22 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the appHcant. 

Defects in the images include but are not limited to the items checked: 



Lfl BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




This Page Blank (uspto) 



